Ethical Considerations in the Design and Conduct of Clinical Trials of Artificial Intelligence

Key Points Question How generalizable are current National Institutes of Health (NIH) ethical principles for conduct of clinical trials to clinical trials of artificial intelligence (AI), and what unique ethical considerations arise in trials of AI? Findings In this qualitative study, interviews with 11 investigators involved in clinical trials of AI for diabetic retinopathy screening confirmed the applicability of current ethical principles but also identified unique challenges, including assessing social value, ensuring scientific validity, fair participant selection, evaluation of risk-to-benefit ratio in underrepresented groups, and navigating complex consent processes. Meaning These results suggest ethical challenges unique to clinical trials of AI, which may provide important guidance for empirical and normative ethical efforts to enhance the conduct of AI clinical trials.


Introduction
The integration of artificial intelligence (AI) into health care promises to address long-standing challenges, offering innovative solutions to improve patient outcomes, health equity, clinician productivity, and system efficiency. 1,24][5][6][7] However, there exists

+ Supplemental content
Author affiliations and article information are listed at the end of this article.
Open Access.This is an open access article distributed under the terms of the CC-BY License.
JAMA Network Open.2024;7(9):e2432482.doi:10.1001/jamanetworkopen.2024  8 This empirical study aims to bridge this gap by examining the practical ethical considerations in the design and implementation of clinical trials involving AI.
Early detection of diabetic retinopathy (DR) is a vanguard area in clinical AI; the first US Food and Drug Administration (FDA) De Novo-authorized autonomous AI was for diabetic eye examinations. 9We collaborated with investigators from the first National Institutes of Health (NIH)funded randomized clinical trial (RCT) of autonomous AI, the AI for Children's Diabetic Eye Exams Study (ACCESS), which was designed to determine the efficacy of autonomous AI screening for DR in a diverse population of youth with diabetes. 102][13] Emanuel et al 12 have further delineated 7 core principles for clinical trial ethics, endorsed by the NIH: social and clinical value, scientific validity, fair participant selection, favorable risk-benefit ratio, independent review, informed consent, and respect for human participants. 149][20][21] While there is a consensus on the necessity for increased transparency of randomized clinical trials of AI (AI-RCTs), current guidelines primarily focus on standardized reporting and fall short from addressing ethical considerations in the design of these clinical trials. 22,23is qualitative study aimed to address 2 primary research questions: (1) To what extent are the 7 NIH ethical principles 14 created by Emanuel and Grady 12 generalizable to clinical trials of AI? and ( 2) What are the ethical considerations that may be unique to clinical trials of AI?

Methods
This qualitative study was approved by the Johns Hopkins Medicine institutional review board.All study participants were informed about the waiver of written consent and provided verbal consent to participate voluntarily, without financial compensation.We followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guidelines.

Study Design
We employed both a deductive and an inductive approach to data collection.We used a deductive approach described to test the applicability of the NIH's 7 core ethical principles-clinical and social value, scientific validity, fair participant selection, favorable risk-benefit ratio, independent review, informed consent, and respect for human participants. 24We also utilized a modified grounded theory approach for the discovery of novel themes. 25

Participants and Recruitment
Participants in this study included clinical investigators, institutional review ethicists, and clinical trialists involved in autonomous AI trials for diabetic retinopathy screening.The selection criteria were aligned with the study's aim of examining ethical challenges in the design and conduct of AI-based clinical trials.Initially, purposive, nonprobabilistic sampling was used to recruit 6 participants from the ACCESS study (NCT05131451).

Themes Overview
While recognizing the importance of the 7 ethical principles in AI clinical trials, participants identified unique ethical challenges specific to AI trials.These challenges demand a nuanced understanding of how to appropriately apply these principles in the context of AI clinical trials.Table 2 outlines participants' perspectives on the 7 principles within the context of AI clinical trials.Table 3 presents novel ethical considerations that emerged from the inductive analysis, highlighting specific challenges faced during the implementation of these trials.The broadly defined term "value" in AI system design is subject to varied interpretations across different stakeholders such as health systems, clinicians, and patients, highlighting conflicts in priorities and concerns about the system's inflexibility to adapt to individual patient needs • "So patient value can take all what people really, really, really care about which includes spiritual and religious issues which do not go into health services research considerations, don't go into societal considerations, don't go into…so there's a lot of…so the very word value I would bet is not welldefined."(participant 3) • "Somebody who had very limited resources couldn't afford to pay $60 a month for drops.…So, the decision that we make about how much quality we can afford needs to be made with respect to local economic scales."(participant 6) Can AI integration enhance clinical workflows without compromising patient safety?
Integrating AI into clinical workflows presents a significant tension between enhancing operational efficiency and the potential risk of complicating existing workflows, with unresolved questions about the actual improvement of clinical outcomes • "I think part of it, to not be able to do an RCT with an AI tool, would come down to just clinical workflow.If you're trying to interject something into a workflow that's already overloaded and pretty strapped.…Ithink that's a limitation when it comes to implementing these because you have to set up, you know, both arms.You have to then set up multiple workflows and you're already…you're already trying to interject a new workflow where it hadn't been before, which can be complicated."(participant 8) • "I think the clinical value is that there's quite a bit of hype about AI and we know that for sure AI can do certain things better than humans in many different contexts, but just because AI is better than humans that do certain things doesn't mean if we incorporate AI will it necessarily improve the outcomes or the metrics that we're interested in.So I think it's very important to in clinical trial involving AI to show that it improves outcomes.That's a completely open question by now that we don't know for the most part whether incorporating AI in clinical workflow improves outcome, so I think this is a very important question to answer.And you could only answer that using randomized clinical trial."(participant 4)

How to balance the economic incentives with the ethical obligations to adopt effective AI interventions that can improve patients' outcomes?
There is a profound conflict between the economic demands of conducting high-cost randomized clinical trials and the ethical imperative to make AI tools universally accessible, raising concerns about health care inequities, especially in lower-income settings • "If you don't get creators and people like me and investors excited about the potential return, it will stop.That's just the way it is.I was struggling as a developer, what is the balance between making…so [that] if you see that more access and better outcomes is good and you expect people to pay for that, how do you put a charge so that you don't make a charge too high?"(participant 7) • "Randomized control trials…there's sort of an inherent assumption that they have to cost 40 million dollars.I just think that's unethical.We have to come up with a way of delivering the kind of evidence, the high-quality evidence that can provide in a way that's affordable for lower middle-income countries."(participant 6) What are the ethical implications of expanding DR screening without enhancing treatment access?
Expanding AI-driven diabetic retinopathy screening to underserved populations without improving access to treatment could create ethical dilemmas, potentially increasing rather than decreasing health disparities • "If someone's not accessing services for diabetes in general regularly and not coming in for their well visits, …they're not controlling their condition in the first place, they're probably coming in less, they're less likely to get the AI screening even if it's available to them in their clinic and they're more at risk for diabetic retinopathy because they're not [accessing care].…Ifwe're trying to target and reduce those disparities, I think it would be important then to see where they're falling off and why they're not getting screened or are they getting screened and not going for follow-ups."(participant 2) • "I've been pleasantly surprised at how well the generalization has shown so far.…ButI also think that if you only train for people from one small part of the world, then you would have a generalization problem."(participant 10) • "If the AI is efficacious, it's an accurate diagnostic, but is this cost-effective if it costs a million dollars to run?Whether a system is willing to pay for such a product is another question."(participant 11) Abbreviations: AI, artificial intelligence; DR, diabetic retinopathy; RCT, randomized clinical trial.

JAMA Network Open | Ethics
Ethical Additionally, concerns were raised about the adequacy of current informed consent processes and institutional ethical review readiness to assess the risks and benefits of AI interventions.

Novel Ethical Considerations in Clinical Trials of AI
Participants highlighted additional ethical challenges beyond the established 7 ethical principles (Table 3).These included: ( what is the balance between making…so if you see that more access and better outcomes is good and you expect people to pay for that, how do you put a charge so that you don't make a charge too high?"The same participant also added insight on payment models for AI, suggesting a focus on health equity."How should we be paying for AI?" he said."If as a taxpayer or society you're paying for something, then health equity should be the main guiding star." One participant in ophthalmology criticized the inherently high costs associated with RCTs: "Randomized control trials…there's sort of an inherent assumption that they have to cost 40 million dollars.I just think that's unethical.We have to come up with a way of delivering the kind of evidence, the high-quality evidence that can provide in a way that's affordable for lower middle-income countries." Finally, the practical implications of AI were discussed."If you're improving the screening, then that in and of itself is sufficient to understand this is something that benefits patients," said a participant working in AI and machine learning in health care."Now whether a system is willing to pay for such an [AI solution], that's a different question."

What Are the Ethical Implications of Expanding DR Screening Without Enhancing Treatment
Access? | A stated goal of AI is to broaden the reach of ophthalmology screening to populations currently underserved, thereby reducing their risk of blindness from DR.However, participants highlighted a complex array of clinical and ethical questions associated with this proposed use of AI.
One concern is whether merely expanding access to DR screening without simultaneously improving access to treatment might create ethical dilemmas downstream.
"If someone's not accessing services for diabetes in general regularly and not coming in for their well visits,…they're not controlling their condition in the first place, they're probably coming in less, they're less likely to get the AI screening even if it's available to them in their clinic and they're more at risk for diabetic retinopathy because they're not [accessing care]," said a researcher specializing in biostatistics and clinical trials."If we're trying to target and reduce those disparities, I think it would be important then to see where they're falling off and why they're not getting screened, or are they getting screened and not going for follow-ups."

Question
How generalizable are current National Institutes of Health (NIH) ethical principles for conduct of clinical trials to clinical trials of artificial intelligence (AI), and what unique ethical considerations arise in trials of AI? Findings In this qualitative study, interviews with 11 investigators involved in clinical trials of AI for diabetic retinopathy screening confirmed the applicability of current ethical principles but also identified unique challenges, including assessing social value, ensuring scientific validity, fair participant selection, evaluation of riskto-benefit ratio in underrepresented groups, and navigating complex consent processes.Meaning These results suggest ethical challenges unique to clinical trials of AI, which may provide important guidance for empirical and normative ethical efforts to enhance the conduct of AI clinical trials.
Considerations in the Design and Conduct of Clinical Trials of AI JAMA Network Open.2024;7(9):e2432482.doi:10.1001/jamanetworkopen.2024.32482(Reprinted) September 6, 2024 5/12 Downloaded from jamanetwork.comby guest on 09/11/2024 Participant Selection | Fair participant selection in AI clinical trials emerged as a significant topic, particularly regarding the accurate representation of the patient population of focus.Study participants highlighted the challenges in evaluating the efficacy of the AI intervention across patient subgroups, who are often affected by limited access to care and can be underrepresented in clinical trials.One participant pointed out the complexities of ensuring equitable access studying the impact of AI screening on patient groups that may access less regular diabetes screening and care.Favorable Risk-Benefit Ratio | Participants recognized the complexity of balancing the risks and benefits of AI interventions across diverse patient groups.They noted the difficulty in estimating the harm-to-benefit ratio of AI interventions relative to the known risks of standard care, a challenge exacerbated by limited representation of patient groups facing health inequities in retrospective standalone studies of algorithm performance.Informed Consent | Participants identified key ethical concerns presented in AI clinical trials, emphasizing the need for transparent communication about the risk and benefits of an AI intervention tailored to patients with varying levels of health literacy.They questioned whether patients fully understand the extent toward which their data might be used beyond the trial itself.
between the extensive theoretical research on ethical concerns in AI applications in health care and the practical challenges encountered by clinical investigators in clinical settings.

JAMA Network Open | Ethics Ethical
Considerations in the Design and Conduct of Clinical Trials of AI 10Two authors (R.W. and D.C.) invited investigators from the ACCESS study to participate in this study.To enhance the generalizability of

Table 2 .
Ethical Considerations for Applying the 7 Ethical Principles to Clinical Trials Involving AI Just because AI is better than humans that do certain things doesn't mean if we incorporate AI it will necessarily improve the outcomes or the metrics that we're interested in.So I think it's very important in clinical trials involving AI to show that it improves outcomes.That's a completely open question by now that we don't know for the most part whether incorporating AI in clinical workflow improves outcome, so I think this is a very important question to answer."(participant4) • "I think we should see what [social value] is from the patient's perspective that would be beneficial and then identify and measure that potentially with a quantitative metric."(participant8) • "Having to do a dilated eye exam is a whole thing in trying to access those services, but if you can provide that first level of screening really easily in a lot of different locations and make it really easy, it's going to ultimately reduce disparities in the disease if it's detected earlier."And that's the big risk, you know, of course we train these things to only work with images that come from super-expensive cameras or come from, you know, the best, most recent x-ray and CT and MRI machines and stuff, then you basically…I said we produced a Cadillac that literally can't be driven on the roads of lower middle income country cities, I mean [it]becomes a real problem.So I think there's an obligation to try to produce the tools openly, the ideal thing is to be producing an AI that is designed for use in low resource • "I think there are some issues with AI around scientific validity because it's more of a systems intervention and to see whether individual randomization, you know, it's hard, right, because for most studies we prefer individual randomization.I think another real problem with AI studies is what is the usual care component and does it make sense to compare to usual care when…usual care can vary tremendously now and what happens when all of a sudden a health system already has an AI, so is this competing, right.Is it comparing one AI to the existing AI?" (participant 1) • "I think my biggest concern about AI is, as I've said at the outset, there's the potential for AI not to lead to greater equity, but actually to exacerbate existing inequity because of the problems you just touched on, that if it's not trained in a particular setting, that it may not have been relevant in that setting.ahead of the study or ahead of the data collection.But yeah, I mean I think [institution] from the beginning wants to ensure that, you know, the ownership of the images is still with the local partner, and then there is potential, you know, a license to use those images in a deidentified manner for training."(participant 5) Abbreviations: AI, artificial intelligence; CT, computed tomography; MRI, magnetic resonance imaging; RCT, randomized clinical trial.JAMA Network Open | Ethics Ethical Considerations in the Design and Conduct of Clinical Trials of AI JAMA Network Open.2024;7(9):e2432482.doi:10.1001/jamanetworkopen.2024.32482(Reprinted) September 6, 2024 4/12 Downloaded from jamanetwork.comby guest on 09/11/2024 Nuanced Considerations

of the 7 Ethical Principles in AI Clinical Trials When
applied to clinical trials of AI in clinical settings, participants identified several unique applications of the 7 ethical principles.Common themes across principles included the added difficulty in accounting for equitable access to care and the need for transparency with patients.

Table 3 .
Novel Ethical Consideration for Conducting Clinical Trials of AI

How to Balance the Economic Incentives With the Ethical Obligations to Adopt Effective AI Intervention That Can Improve Patients' Outcomes? |
Ethical Considerations in the Design and Conduct of Clinical Trials of AI limitation when it comes to implementing these because you have to set up, you know, both arms.You have to then set up multiple workflows and you're already…you're already trying to interject a new workflow where it hadn't been before, which can be complicated.""Ithink the clinical value is that there's quite a bit of hype about AI and we know that for sure AI can do certain things better than humans in many different contexts, but just because AI is better than humans that do certain things doesn't mean if we incorporate AI will it necessarily improve the outcomes or the metrics that we're interested in," said a respondent specializing in ophthalmology and machine learning."SoI think it's very important to, in clinical trial[s] involving AI, to show that it improves outcomes.That's a completely open question by now that we don't know for the most part whether incorporating AI in clinical workflow improves outcome, so I think this is a very important question to answer.And you could only answer that using [a] randomized clinical trial."Participantshighlighted a critical tension in developing and validating AI tools in health care, balancing economic pressures with ethical imperatives.The ethical mandate to make these tools universally accessible clashes with the high costs associated with conducting RCTs, deemed the criterion standard for validation, particularly in the US and Europe.This economic challenge has prompted AI developers to shift RCTs to developing countries, raising concerns about potentially deepening health care inequities.One participant who is both an AI developer and clinician-scientist expressed the dilemma facing AI developers and health systems: "If you don't get creators and people like me and investors excited about the potential return, it will stop.That's just the way it is.I was struggling as a developer, still unclear.This tension is further amplified when contemplating potential risks to research participants during clinical trials."I think part of it, to not be able to do an RCT with an AI tool, would come down to just clinical workflow," said a participant working in optometry."If you're trying to interject something into a workflow that's already overloaded and pretty strapped.…Ithink that's a JAMA Network Open | Ethics JAMA Network Open.2024;7(9):e2432482.doi:10.1001/jamanetworkopen.2024.32482(Reprinted) September 6, 2024 6/12 Downloaded from jamanetwork.comby guest on 09/11/2024